feat: packagist worker#4357
Conversation
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
PR SummaryMedium Risk Overview Data & workflows: Migration creates Implementation: New Reviewed by Cursor Bugbot for commit 282c403. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
|
There was a problem hiding this comment.
Pull request overview
Adds a Packagist worker that discovers Composer packages and ingests metadata, dependencies, maintainers, repositories, and download statistics.
Changes:
- Adds Packagist Temporal workflows, schedules, persistence, and deployment wiring.
- Adds DAL queries and ingestion-state migration.
- Adds documentation and unit tests.
Reviewed changes
Copilot reviewed 41 out of 43 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
services/libs/data-access-layer/src/packages/versions.ts |
Adds Packagist version upserts. |
services/libs/data-access-layer/src/packages/packagistPackageState.ts |
Adds ingestion watermark queries. |
services/libs/data-access-layer/src/packages/packages.ts |
Adds Packagist package DAL operations. |
services/libs/data-access-layer/src/packages/index.ts |
Exports new DAL modules. |
services/libs/data-access-layer/src/packages/dependencies.ts |
Adds dependency-edge upserts. |
services/apps/packages_worker/vitest.config.ts |
Isolates Packagist tests. |
services/apps/packages_worker/src/workflows/index.ts |
Exports Packagist workflows. |
services/apps/packages_worker/src/security-contacts/extractors/registry/index.ts |
Registers Packagist contact extraction. |
services/apps/packages_worker/src/scripts/triggerPackagistSeed.ts |
Adds manual workflow trigger. |
services/apps/packages_worker/src/packagist/workflows.ts |
Defines ingestion workflows. |
services/apps/packages_worker/src/packagist/upsertPackageInfo.ts |
Persists package information. |
services/apps/packages_worker/src/packagist/upsertMetadata.ts |
Persists versions and dependencies. |
services/apps/packages_worker/src/packagist/types.ts |
Defines Packagist types. |
services/apps/packages_worker/src/packagist/schedule.ts |
Registers Temporal schedules. |
services/apps/packages_worker/src/packagist/retryPolicy.ts |
Defines retry limit. |
services/apps/packages_worker/src/packagist/README.md |
Documents worker behavior. |
services/apps/packages_worker/src/packagist/normalize.ts |
Normalizes registry metadata. |
services/apps/packages_worker/src/packagist/listPackages.ts |
Fetches and parses package catalog. |
services/apps/packages_worker/src/packagist/fetchPackage.ts |
Fetches package endpoints. |
services/apps/packages_worker/src/packagist/expandMetadata.ts |
Expands minified metadata. |
services/apps/packages_worker/src/packagist/downloads.ts |
Persists 30-day downloads. |
services/apps/packages_worker/src/packagist/activities.ts |
Implements Temporal activities. |
services/apps/packages_worker/src/packagist/__tests__/wiring.test.ts |
Tests worker wiring. |
services/apps/packages_worker/src/packagist/__tests__/persistMetadata.test.ts |
Tests metadata persistence. |
services/apps/packages_worker/src/packagist/__tests__/normalize.test.ts |
Tests normalization. |
services/apps/packages_worker/src/packagist/__tests__/listPackages.test.ts |
Tests catalog handling. |
services/apps/packages_worker/src/packagist/__tests__/ingest.test.ts |
Tests ingestion behavior. |
services/apps/packages_worker/src/packagist/__tests__/fetchPackage.test.ts |
Tests HTTP handling. |
services/apps/packages_worker/src/packagist/__tests__/expandMetadata.test.ts |
Tests metadata expansion. |
services/apps/packages_worker/src/packagist/__tests__/dueSelection.test.ts |
Tests due-selection queries. |
services/apps/packages_worker/src/packagist/__tests__/downloads.test.ts |
Tests download windows. |
services/apps/packages_worker/src/bin/packagist-worker.ts |
Adds worker entrypoint. |
services/apps/packages_worker/src/activities.ts |
Exports Packagist activities. |
services/apps/packages_worker/package.json |
Adds worker scripts. |
scripts/services/packagist-worker.yaml |
Adds local service configuration. |
scripts/builders/packages.env |
Adds worker build target. |
docs/adr/README.md |
Indexes Packagist ADR. |
docs/adr/0006-packagist-worker-design-decisions.md |
Records design decisions. |
docs/adr/0005-pypi-downloads-bigquery-merge-scoping.md |
Expands decider name. |
backend/src/osspckgs/migrations/V1783382400__packagist_worker.sql |
Creates ingestion-state table. |
backend/.env.dist.local |
Adds local crawler contact setting. |
backend/.env.dist.composed |
Adds composed crawler contact setting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 49 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
docs/adr/0009-packagist-worker-design-decisions.md:33
- This new ADR does not follow the mandatory repository ADR structure.
.claude/rules/adr-format.md:18-28requires## Decision,## Alternatives Considered, and## Consequencesin that order (with Positive/Negative/Risks subsections), but this document uses a living## Decisionslayout and omits the alternatives section. Please restructure it to the enforced template or split the consolidated decisions into compliant ADRs.
## Decisions
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 49 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
docs/adr/0009-packagist-worker-design-decisions.md:33
- This new ADR does not follow
.claude/rules/adr-format.md: the mandatory heading is## Decision(singular), followed by## Alternatives Consideredand## Consequenceswith### Positive,### Negative, and### Risks. Those required sections are absent or use a different structure, so please reformat the ADR before merging.
## Decisions
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 49 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
services/apps/packages_worker/src/packagist/activities.ts:225
- A successful response without a numeric
downloads.monthlycurrently becomes a no-op and still advancesdownloads_30d_last_run_at. Because this lane has no backfill and the month is never revisited, a partial/malformed response permanently loses that month's observation. Treat the missing count as an error (so Temporal retries and eventually records an error) and only mark success after a numeric value is persisted.
await persistPackagist30dWindow(qx, purl, info.value.package.downloads?.monthly ?? null, runDate)
await markPackagist30dProcessed(qx, purl, { status: 'success', attempts: info.attempts })
services/libs/data-access-layer/src/packages/versions.ts:165
- This cleanup can change
versions.is_latest, but its row count is discarded, sochangedFieldsmay omit the only mutation (for example, when the incoming latest was already true while another stale row was also true).persistPackagistMetadatathen writes no audit token for the cleanup. Capture the affected-row count and addversions.is_latestwhen any stale flag was cleared.
docs/adr/0009-packagist-worker-design-decisions.md:33 - This ADR does not follow the repository's mandatory ADR structure.
.claude/rules/adr-format.md:18-28requires## Decision,## Alternatives Considered, and## Consequenceswith### Positive,### Negative, and### Risks; this file uses## Decisions, has no alternatives section, and embeds differently named consequence blocks per decision. Restructure it to the required template or update the governing rule before accepting the ADR.
## Decisions
services/apps/packages_worker/src/packagist/activities.ts:270
- When
downloads.dailyis absent or non-numeric, this records the run as successful even though no row was inserted. The next schedule uses a new run date, so the missing day is never retried or backfilled. Treat the missing count as an error and only advance the daily watermark after a numeric count has been persisted.
await markPackagistDailyProcessed(qx, candidate.purl, {
status: 'success',
attempts: info.attempts,
})
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 49 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
docs/adr/0009-packagist-worker-design-decisions.md:33
- This ADR does not follow the repository's mandatory ADR structure:
.claude/rules/adr-format.md:18-28requires## Decision,## Alternatives Considered, and## Consequenceswith### Positive,### Negative, and### Risks. The current consolidated## Decisionslayout omits those required sections, so it must be restructured or the rule must explicitly define an exemption for living ADRs.
## Decisions
Signed-off-by: anilb <epipav@gmail.com>
Signed-off-by: anilb <epipav@gmail.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 49 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (2)
services/apps/packages_worker/src/packagist/activities.ts:339
- This makes the nominal weekly refresh run approximately every two weeks. If run 1 starts at
t0, its rows are stamped aftert0; run 2 starts att0 + 7d, computesdueCutoff = t0, and the strictmetadata_last_run_at < dueCutoffpredicate skips every row written during run 1. Please either select all rows last processed before the current run cutoff for the seed-triggered weekly sweep, or schedule the sweep more frequently and retain the seven-day rolling threshold.
const dueCutoff = new Date(
new Date(cutoff).getTime() - metadataRefreshDays() * 24 * 60 * 60 * 1000,
).toISOString()
docs/adr/0009-packagist-worker-design-decisions.md:33
- This new ADR does not follow the repository's mandatory ADR structure.
.claude/rules/adr-format.md:18-28requires## Decision,## Alternatives Considered, and## Consequenceswith### Positive,### Negative, and### Risks; this document uses## Decisions, has no alternatives section, and nests consequence labels differently. Please restructure it or update the ADR policy separately before adding this accepted ADR.
## Decisions
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 49 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (3)
services/apps/packages_worker/src/packagist/activities.ts:339
- With the default 7-day window and weekly seed schedule, this threshold skips every package processed after the previous run's start: its
metadata_last_run_at = NOW()is later thanthis cutoff - 7 days. Early packages therefore refresh nearly every 14 days, not weekly. Record the fixed workflow cutoff as the watermark (or otherwise compare run anchors) so one run's completion timestamps are eligible on the next weekly run.
const dueCutoff = new Date(
new Date(cutoff).getTime() - metadataRefreshDays() * 24 * 60 * 60 * 1000,
).toISOString()
services/apps/packages_worker/src/packagist/activities.ts:138
- A terminal
NOT_FOUNDonly advances the state watermark, leaving a deleted Packagist package permanentlystatus = 'active'. Because seeding is insert-only, nothing else will mark that row unpublished, and it remains in future crawls and downstream active-package data. Handle persistentNOT_FOUNDby updating the package status while retaining the row for history.
await markPackagistMetadataScanned(
docs/adr/0009-packagist-worker-design-decisions.md:33
- This new ADR does not follow the repository's mandatory ADR structure:
.claude/rules/adr-format.md:18-28requires## Decision,## Alternatives Considered, and## Consequenceswith### Positive,### Negative, and### Risks. The plural## Decisionsplus inline consequence labels does not satisfy those headings; please restructure the document or update the governing rule first.
## Decisions
|
|
||
| // Match: alpha|a|beta|b|rc followed by optional digits | ||
| // Stable: plain number, -patch\d*, -p\d* | ||
| if (suffix.match(/^(alpha|a|beta|b|rc)\d*$/)) { |
| }) | ||
|
|
||
| const suffixMatch = | ||
| dashIdx === -1 ? null : v.slice(dashIdx + 1).match(/^(alpha|a|beta|b|rc)(\d*)/i) |
|
|
||
| let versionIds: Array<{ number: string; id: string }> = [] | ||
| if (versionRows.length > 0) { | ||
| const versionResult = await upsertPackagistVersions(t, agg.id, versionRows, latestVersion) |
No description provided.